Ollama

So simple school kids can do it!

Johannes B. Gruber

Contents

  • Motivation
  • What is Ollama
  • Why use generative Larger Language Models?
  • Why use models locally?

Late 2022

Question: How can you run generative large language models (gLLM; aka GPT, aka ChatGPT-like, aka GPT-like, aka generative AI, AI text bots) locally?

Late 2023

Question: How can you run generative large language models (gLLM; aka GPT, aka ChatGPT-like, aka GPT-like, aka generative AI, AI text bots) locally?

What is Ollama?

Ollama: a tool/framework

  • Ollama is a tool for running large language models (LLMs) locally
  • It optimizes setup and configuration details, including GPU usage
  • Is easy to install and run
  • Easy to extend
  • MIT license (aka. open-source)!

Ollama models

  • Ollama bundles model weights, configuration, and data into a single package, defined by a Modelfile
  • A curated model library is available at https://ollama.ai/library
  • Custom models can be up/downloaded from https://ollamahub.com
  • Huggingface models can be imported (Llama, Mistral, Falcon, RW and BigCode architecture supported atm)
Model Parameters Size
Llama 2 7B 3.8GB
Mistral 7B 4.1GB
Dolphin Phi 2.7B 1.6GB
Phi-2 2.7B 1.7GB
Neural Chat 7B 4.1GB
Starling 7B 4.1GB
Code Llama 7B 3.8GB
Llama 2 Uncensored 7B 3.8GB
Llama 2 13B 13B 7.3GB
Llama 2 70B 70B 39GB
Orca Mini 3B 1.9GB
Vicuna 7B 3.8GB
LLaVA 7B 4.5GB

Why use gLLMs?

  • cheaper to use 💸 (experts > research assistants > crowd workers > gLLMs)
  • more accurate 🧠 (experts > research assistants > gLLMs > crowd workers)
  • feasible to make large training set (for SML) or code entire corpus
  • multilingual1
  • better at annotate complex categories 🤔 (human* > gLLM > LLM > SML)
  • can use contextual information (human* > gLLM > LLM > ❌ SML)
  • easier to use: plain English prompting

(e.g., Gilardi, Alizadeh, and Kubli 2023; Heseltine and Hohenberg 2023; Zhong et al. 2023; Törnberg 2023)

Why is local important?

Why should we run gLLMs locally?

  • No guardrails 🤨

Why should we run gLLMs locally?

  • No guardrails 🤨

Why should we run gLLMs locally?

  • privacy (nothing is uploaded anywhere)
  • cost (cheap and reliable pricing)
  • long-term reproducible
  • other ethical concerns (?)
  • transparency (?)

(Weber and Reichardt 2023; Spirling 2023)

Examples

Classification

Classification–Strategies

Prompting strategies
Prompting Strategy Example Structure
Zero-shot {"role": "system", "content": "Text of System Prompt"},
{"role": "user", "content": "(Text to classify) + classification question"}
One-shot {"role": "system", "content": "Text of System Prompt"},
{"role": "user", "content": "(Example text) + classification question"},
{"role": "assistant", "content": "Example classification"},
{"role": "user", "content": "(Text to classify) + classification question"}
Few-shot {"role": "system", "content": "Text of System Prompt"},
{"role": "user", "content": "(Example text) + classification question"},
{"role": "assistant", "content": "Example classification"},
{"role": "user", "content": "(Example text) + classification question"},
{"role": "assistant", "content": "Example classification"},
. . . more examples
{"role": "user", "content": "(Text to classify) + classification question"}
Chain-of-Thought {"role": "system", "content": "Text of System Prompt"},
{"role": "user", "content": "(Text to classify) + reasoning question"},
{"role": "assistant", "content": "Reasoning"},
{"role": "user", "content": "Classification question"}

See: Weber and Reichardt (2023)

Classification–Zero-shot

library(rollama)
library(tibble)
library(purrr)
q <- tribble(
  ~role,    ~content,
  "system", "You assign texts into categories. Answer with just the correct category.",
  "user",   "text: the pizza tastes terrible\ncategories: positive, neutral, negative"
)
query(q)
#> 
#> ── Answer ────────────────────────────────────────────────────────
#> Category: Negative

Also packages for Dart, Swift, C#, Java, PHP, Rust etc.

Python:

import ollama
response = ollama.chat(model='llama2', messages=[
  {
    'role': 'system',
    'content': 'You assign texts into categories. Answer with just the correct category.',
  },
  {
    'role': 'user',
    'content': 'text: the pizza tastes terrible\ncategories: positive, neutral, negative',
  },
])
print(response['message']['content'])
#> Category: Negative

JavaScript:

import ollama from 'ollama'

const response = await ollama.chat({
  model: 'llama2',
  messages: [
    {
      'role': 'system',
      'content': 'You assign texts into categories. Answer with just the correct category.',
    },
    {
      'role': 'user',
      'content': 'text: the pizza tastes terrible\ncategories: positive, neutral, negative',
    },
  ],
})
console.log(response.message.content)
#> Category: Negative

Classification–One-shot

q <- tribble(
  ~role,    ~content,
  "system", "You assign texts into categories. Answer with just the correct category.",
  "user", "text: the pizza tastes terrible\ncategories: positive, neutral, negative",
  "assistant", "Category: Negative",
  "user", "text: the service is great\ncategories: positive, neutral, negative"
)
query(q)
#> 
#> ── Answer ────────────────────────────────────────────────────────
#> Category: Positive

Neat effect: change the output structure

q <- tribble(
  ~role,    ~content,
  "system", "You assign texts into categories. Answer with just the correct category.",
  "user", "text: the pizza tastes terrible\ncategories: positive, neutral, negative",
  "assistant", "{'Category':'Negative','Confidence':'100%','Important':'terrible'}",
  "user", "text: the service is great\ncategories: positive, neutral, negative"
)
answer <- query(q)
#> 
#> ── Answer ────────────────────────────────────────────────────────
#> {'Category':'Positive','Confidence':'100%','Important':'great'}

Classification–Few-shot

q <- tribble(
  ~role,    ~content,
  "system", "You assign texts into categories. Answer with just the correct category.",
  "user", "text: the pizza tastes terrible\ncategories: positive, neutral, negative",
  "assistant", "Category: Negative",
  "user", "text: the service is great\ncategories: positive, neutral, negative",
  "assistant", "Category: Positive",
  "user", "text: I once came here with my wife\ncategories: positive, neutral, negative",
  "assistant", "Category: Neutral",
  "user", "text: I once ate pizza\ncategories: positive, neutral, negative"
)
query(q)
#> 
#> ── Answer ────────────────────────────────────────────────────────
#> Category: Neutral

Classification–Chain-of-Thought

q_thought <- tribble(
  ~role,    ~content,
  "system", "You assign texts into categories. ",
  "user",   "text: the pizza tastes terrible\nWhat sentiment (positive, neutral, or negative) would you assign? Provide some thoughts."
)
output_thought <- query(q_thought)
#> 
#> ── Answer ────────────────────────────────────────────────────────
#> 
#> Based on the given text, I would assign it to the category of "negative
#> sentiment." The use of the word "terrible" and the tone of the sentence convey
#> a strong negative emotion towards the taste of the pizza. It's unlikely that
#> someone would describe something as "terrible" if they were neutral or
#> positive about it.

Now we can use these thoughts in classification

q <- tribble(
  ~role,    ~content,
  "system", "You assign texts into categories. ",
  "user",   "text: the pizza tastes terrible\nWhat sentiment (positive, neutral, or negative) would you assign? Provide some thoughts.",
  "assistant", pluck(output_thought, "message", "content"),
  "user",   "Now answer with just the correct category (positive, neutral, or negative)"
)
query(q)
#> 
#> ── Answer ────────────────────────────────────────────────────────
#> 
#> Negative

Full tutorial (inlcuding how to batch annotate): https://jbgruber.github.io/rollama/articles/annotation.html

Annotating news events

tinytable_a84l16f94w544insk9yz
cluster url date title
1 https://www.thesun.ie... 2023-12-21 22:59:03 Arsenal and Liverpool only two of Prem’s Big Six NOT to reject new Super League plot as football descends i...
1 https://www.rte.ie/sp... 2023-12-21 13:29:36 What does the ECJ Super League ruling mean?
1 https://news.sky.com/... 2023-12-21 08:34:00 European Super League: EU's top court rules FIFA and UEFA acted unlawfully in blocking breakaway competition
1 https://news.sky.com/... 2023-12-21 11:48:00 How would the new European Super League work - and what does it mean for Premier League clubs?
1 https://www.breakingn... 2023-12-21 10:06:12 Judges rule Uefa blocking European Super League was contrary to EU law
1 https://www.theguardi... 2023-12-21 09:01:49 European Super League project gets huge boost after court of justice ruling
1 https://www.thejourna... 2023-12-21 11:21:07 Uefa and Fifa bid to block Super League broke EU law, court rules
1 https://www.theguardi... 2023-12-20 19:25:11 Uefa v European Super League: what could verdict mean for football?
1 https://www.thesun.ie... 2023-12-21 09:09:45 Terrified fans fear European Super League is about to return after European Court of Justice ruling...
1 https://www.thesun.ie... 2023-12-21 11:17:45 European Super League LIVE: Uefa and Fifa lose court case as hated tournament is back on the table – l...
Annotation 
library(rollama)
eval_set_df_annotated <- eval_set_df |> 
  group_by(cluster) |> 
  summarise(headlines = paste0(title, collapse = "\n\n"),
            n = length(title)) |> 
  filter(n > 1) |> 
  mutate(label = NA)

for (i in seq_len(nrow(eval_set_df_annotated))) {
  
  q <- tibble::tribble(
    ~role,    ~content,
    "system", "You assign short highly descriptive topic labels to a collection of headlines about the same topic. Answer with just the topic label.",
    
    "user", "headlines: \nGuns, ammunition, drugs and cash seized as part of investigation into organised crime in Tipperary as...\n\nFirearms, ammunition, machete, crack cocaine, heroin and cash seized in garda operation - Irish Mirror Online\n\nMan arrested after gardaí seize weapons and drugs in Clonmel",
    
    "assistant", "blow against organised crime",
    
    "user", "headlines: Taoiseach Leo Varadkar stung by Russian pranksters in hoax diplomatic call\n\nTaoiseach Leo Varadkar ‘pranked’ in phone call by pro-Kremlin Russian ‘comedians’ pretending to be African diplomats\n\nGovernment changes meetings protocol after Taoiseach duped in Russian hoax call\n\nTaoiseach Leo Varadkar ‘pranked’ in phone call by pro-Kremlin Russian ‘comedians’ pretending to be African diplomats\n\nLeo Varadkar falls victim to a prank by two Russian comedians\n\nLeo Varadkar insists he 'played along' with Russian pranksters on hoax video call - Irish Mirror Online\n\nLeo Varadkar falls victim to prank call by pair of Russian comedians - Irish Mirror Online\n\nTaoiseach was 'suspicious immediately' of prank by two Russian comedians but had to play along\n\nRed-faced Leo Varadkar makes urgent change to Government video call security protocols after embarrassing...\n\nI played along with prank call by Russian comedians – Taoiseach\n\nTaoiseach Leo Varadkar ‘played along’ with Russian comedians’ prank phone call after he became suspicious\n\nTaoiseach Leo Varadkar ‘played along’ with Russian comedians’ prank phone call after he became suspicious\n\nLeo Varadkar “pranked” by pro-Kremlin Russian “comedians”",
    
    "assistant", "Taoiseach pranked by Russians",
    
    "user", "headlines: Social media platform X back online after global outage\n\nSocial media platform X back online after global outage\n\nX is down: Thousands of users experiencing outages on platform formerly known as Twitter\n\nTwitter down: X website owned by Elon Musk suffers outages as thousands of users report problems...\n\nElon Musk’s X back online after global outage\n\nSocial media platform X back online after global outage · TheJournal.ie\n\nX users experiencing widespread outages",
    
    "assistant", "platform X outage",
    
    "user", glue::glue("headlines: {eval_set_df_annotated$headlines[i]}")
  )
  eval_set_df_annotated$label[i] <- query(q, screen = FALSE) |> pluck("message", "content")
}

Annotating news events

Classification–Cluster 1: Super League

Annotating news events

Cluster 10: Trump banned from Colorado ballot

Annotating news events

tinytable_9cmxnflaxrrxsl3qd3kg
cluster headlines n label good
1 Arsenal and Liverpool only two of Prem’s Big Si... 14 Super League ✔
2 Taoiseach Leo Varadkar stung by Russian prankst... 13 Taoiseach pranked ✔
3 Ireland takes legal action over UK move to bloc... 13 Ireland v UK ✔
4 Ship carrying cocaine worth tens of millions of... 11 cocaine seizure ✔
5 Urgent warning as 90kmh gusts to hit Ireland as... 11 winds winds warning
6 Sale of vapes to under 18s banned from tomorrow... 11 vaping ban ✔
7 Piers Morgan says Mary Earps didn’t ‘deserve’ B... 10 Mary Earps criticism ✔
8 Boy and girl found guilty of ‘frenzied and fero... 10 murder of Brianna Ghey ✔
9 'Co-conspirator' James Flynn jailed over steali... 9 car theft ✔
10 Trump banned from Colorado ballot in historic r... 8 Trump banned from Colorado ballot ✔
11 Luke Littler, 16, is teen darts star who celebr... 8 teen darts star ✔
12 Munster's Casey sees 'human side' to Snyman mov... 8 rugby transfer ✔
13 Minister rejects Garda Commissioner claims forc... 8 asylum seeker housing ✔
14 Lynn's 'secret deal' allegations left court 'st... 8 Lynn trial ✔
15 Social media platform X back online after globa... 7 X outage ✔
16 Ex-NYC mayor Rudy Giuliani files for bankruptcy... 7 Rudy Giuliani bankruptcy ✔
17 Mary Lou McDonald slams Robbie Keane for Israel... 7 Mary Lou McDonald criticizes Robbie Keane ✔
18 Teenager dies in Sligo crash · TheJournal.ie T... 6 Sligo crash ✔
19 Scramblers, quad bikes, imitation firearm, drug... 6 scramblers ✔
20 'Dead and injured' in Prague shooting Czech po... 5 shooting in Prague ✔
21 No public sector pay deal before Christmas afte... 5 public sector pay negotiations ✔
22 Jurgen Klopp suggests some Liverpool fans shoul... 5 fans criticized football fans criticized
23 Reduced speed limits, climate action on Cabinet... 5 traffic laws ✔
24 Evil Jozef Puska’s wife & two brothers among fi... 5 Ashling Murphy trial ✔
25 Luton captain Tom Lockyer discharged from hospi... 5 cardiac arrest Tom Lockyer cardiac arrest
26 High Court approves €1.9 million settlement in ... 5 settlement approved ✔
27 President Michael D Higgins thanks migrants who... 5 culture enrichment ✔
28 Varadkar questions Sinn Féin plan to drive down... 5 housing market ✔
29 UK weather: Storm Pia warning forces schools to... 4 uk storm ✔
30 Police launch murder investigation after child ... 4 murder investigation ✔

Text embedding

embed_text(text = reviews$full_text[1:3])
#> ✔ embedded 3 texts [4s] 
#> # A tibble: 3 × 4,096
#>   dim_1  dim_2 dim_3  dim_4 dim_5  dim_6  dim_7  dim_8 dim_9 dim_10
#>   <dbl>  <dbl> <dbl>  <dbl> <dbl>  <dbl>  <dbl>  <dbl> <dbl>  <dbl>
#> 1  1.85 -1.71   1.47  0.478 -1.75  0.771  3.01   0.961 1.65   0.569
#> 2  1.14 -3.61   2.10 -0.385 -4.11 -3.09   0.990 -1.06  2.55   1.84 
#> 3 -3.35  0.172 -3.49 -0.569 -3.14  1.25  -0.102  1.15  0.575 -2.33 
#> # ℹ 4,086 more variables: dim_11 <dbl>, dim_12 <dbl>,
#> #   dim_13 <dbl>, dim_14 <dbl>, dim_15 <dbl>, dim_16 <dbl>,
#> #   dim_17 <dbl>, dim_18 <dbl>, dim_19 <dbl>, dim_20 <dbl>,
#> #   dim_21 <dbl>, dim_22 <dbl>, dim_23 <dbl>, dim_24 <dbl>,
#> #   dim_25 <dbl>, dim_26 <dbl>, dim_27 <dbl>, dim_28 <dbl>,
#> #   dim_29 <dbl>, dim_30 <dbl>, dim_31 <dbl>, dim_32 <dbl>,
#> #   dim_33 <dbl>, dim_34 <dbl>, dim_35 <dbl>, dim_36 <dbl>, …

Tutorial for SML: https://jbgruber.github.io/rollama/articles/text-embedding.html

Demo

What I will use here

version: '3.6'

services:

  # ollama and API
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    pull_policy: missing
    tty: true
    restart: unless-stopped
    # Expose Ollama API outside the container stack
    ports:
      - 11434:11434
    volumes:
      - ollama:/root/.ollama
    # GPU support (turn off by commenting with # if you don't have an nvidia gpu)
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities:
                - gpu

  # webui, nagivate to http://localhost:3000/ to use
  ollama-webui:
    image: ghcr.io/ollama-webui/ollama-webui:main
    container_name: ollama-webui
    pull_policy: missing
    volumes:
      - ollama-webui:/app/backend/data
    depends_on:
      - ollama
    ports:
      - 3000:8080
    environment:
      - "OLLAMA_API_BASE_URL=http://ollama:11434/api"
    extra_hosts:
      - host.docker.internal:host-gateway
    restart: unless-stopped

volumes:
  ollama: {}
  ollama-webui: {}

tinyurl.com/ollama-gist

What I will use here

tinyurl.com/r-ollama

Issues

Methodological:

  • field is still an academic Wild West – lack of established standards for the reliable, reproducible and ethical use of gLLMs (see Törnberg draft)
  • open models still lack behind proprietary ones (but not as far as we though a year ago!)
  • consistent bias (as opposed to sporadic bias with a group of human annotators)

Technical:

References

Gilardi, Fabrizio, Meysam Alizadeh, and Maël Kubli. 2023. “ChatGPT Outperforms Crowd Workers for Text-Annotation Tasks.” Proceedings of the National Academy of Sciences 120 (30). https://doi.org/10.1073/pnas.2305016120.
Heseltine, Michael, and Bernhard Clemm von Hohenberg. 2023. “Large Language Models as a Substitute for Human Experts in Annotating Political Text.” OSF. https://doi.org/10.31219/osf.io/cx752.
Spirling, Arthur. 2023. “Why Open-Source Generative AI Models Are an Ethical Way Forward for Science.” Nature 616 (7957): 413–13. https://doi.org/10.1038/d41586-023-01295-4.
Törnberg, Petter. 2023. “ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning.” arXiv. http://arxiv.org/abs/2304.06588.
Weber, Maximilian, and Merle Reichardt. 2023. “Evaluation Is All You Need. Prompting Generative Large Language Models for Annotation Tasks in the Social Sciences. A Primer Using Open Models.” https://arxiv.org/abs/2401.00284.
Zhong, Qihuang, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2023. “Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-Tuned BERT.” arXiv. http://arxiv.org/abs/2302.10198.